Predicting the effectiveness of the online clinical clerkship curriculum: Development of a multivariate prediction model and validation study

Given scientific and technological advancements, expectations of online medical education are increasing. However, there is no way to predict the effectiveness of online clinical clerkship curricula. To develop a prediction model, we conducted cross-sectional national surveys in Japan. Social media surveys were conducted among medical students in Japan during the periods May–June 2020 and February–March 2021. We used the former for the derivation dataset and the latter for the validation dataset. We asked students questions in three areas: 1) opportunities to learn from each educational approach (lectures, medical quizzes, assignments, oral presentations, observation of physicians’ practice, clinical skills practice, participation in interprofessional meetings, and interactive discussions with physicians) in online clinical clerkships compared to face-to-face, 2) frequency of technical problems on online platforms, and 3) satisfaction and motivation as outcome measurements. We developed a scoring system based on a multivariate prediction model for satisfaction and motivation in a cross-sectional study of 1,671 medical students during the period May–June 2020. We externally validated this scoring with a cross-sectional study of 106 medical students during February–March 2021 and assessed its predictive performance. The final prediction models in the derivation dataset included eight variables (frequency of lectures, medical quizzes, oral presentations, observation of physicians’ practice, clinical skills practice, participation in interprofessional meetings, interactive discussions with physicians, and technical problems). We applied the prediction models created using the derivation dataset to a validation dataset. The prediction performance values, based on the area under the receiver operating characteristic curve, were 0.69 for satisfaction (sensitivity, 0.50; specificity, 0.89) and 0.75 for motivation (sensitivity, 0.71; specificity, 0.85). We developed a prediction model for the effectiveness of the online clinical clerkship curriculum, based on students’ satisfaction and motivation. Our model will accurately predict and improve the online clinical clerkship curriculum effectiveness.

Introduction Given scientific and technological advancements, the medical education community's expectations of online education are rising [1][2][3]. In addition, coronavirus disease 2019 (COVID- 19) has necessitated the use of the online platform in medical education [4][5][6][7][8]. Globally, online clinical clerkship was proposed to replace face-to-face clinical clerkship during the COVID-19 pandemic (mid-COVID-19) [9]. Various opinions and practical innovations have been reported in relation to making online clinical clerkship a viable alternative to face-to-face clerkship [10][11][12]. However, there are currently no scores that can predict the effectiveness of online clinical clerkship curricula. Developing such predictive scores would make online clinical clerkship more effective.
We aimed to develop and externally validate a prognostic score that represents students' satisfaction and motivation, and can predict the effectiveness of the online clinical clerkship curriculum.

Survey design and data collection
This study followed the principles of the Declaration of Helsinki and was approved by Kyushu University's ethics committee. To achieve our objectives, we conducted cross-sectional surveys. We developed the survey questions with a focus on content validity through an iterative process within the team, which included a medical educator. In addition, we administered the survey to a pilot group of 30 medical students. During a semi-structured interview after the survey was administered, these students were asked to provide feedback on the clarity of formulation of the items and on the format of the questionnaire.
The study consisted of two parts. First, we collected the survey responses for the derivation dataset. Through derivation analysis, we developed a scoring model for the online curriculum to predict medical students' satisfaction and motivation. We then collected the survey responses for the validation dataset. Developed using the derivation dataset, we assessed our predictive model's prediction performance via validation analysis.
The study participants were undergraduate medical school students in Japan. The survey that yielded the derivation dataset was conducted from May 29, 2020 to June 14, 2020. This period was chosen because the first declaration of a state of emergency due to COVID-19 in Japan (April 7-May 25, 2020) led to the widespread use of online clinical clerkship. We conducted the survey that yielded the validation dataset from February 26 to March 11, 2021. This period was chosen because the second declaration of a state of emergency due to COVID-19 in Japan (January 7-March 21, 2021) also led to the widespread use of online clinical clerkship.
Both surveys have identical content. The questionnaire was presented along with an informative letter summarizing the research purpose, as well as the informed consent form, written briefly and clearly in Japanese to avoid misinterpretation. The questionnaire took 3-5 min to complete.
The inclusion criteria were as follows: students (1) who switched from face-to-face to online clinical clerkship mid-COVID-19 and (2) completed all the survey questions. We included fifth-and sixth-year medical students in the derivation dataset and fourth-and fifth-year students in the validation study. This is because in Japan, medical students graduate in March, so sixth-year students often do not participate in clinical clerkships between January and March. The Japanese undergraduate medical education curriculum is six years period. It usually consists of four/three and half years of preclinical education and two/two and half years of clinical clerkship. Clinical clerkship has medical students participate as members of a medical team in a variety of clinical settings. Medical students can acquire clinical competencies through clinical clerkship.

Survey items
The questionnaire asked respondents to share information about their demographic information. If participants were medical students who switched from face-to-face to online clinical clerkship, we inquired as to the effectiveness of online compared to face-to-face clinical clerkship and, how frequently they had various experiences while completing their online clinical clerkship.

Exposure factors and other factors
We examined the extent of students' exposure to factors associated with the effectiveness of online clinical clerkship. We decided which factors should be included in our survey based on previous literature and recommendations for medical education. The factors incorporated into this study included lecture duration [13][14][15] and lecture frequency per week [13,16,17]; opportunities to take quizzes [18,19], submit assignments [20,21], give oral presentations [22,23], observe physicians' practice [10,24], practice clinical skills [10,24], participate in interprofessional meetings [24][25][26], and interact with physicians [24,27]; and the frequency of technical Internet-related problems [28]. For details regarding the options for each answer, see S1 Method

Measuring outcomes
Based on Kirkpatrick's assessment model for assessing the usefulness of medical education and medical curricula [26,[29][30][31], we used two outcome measures of the effectiveness of online clinical clerkship: satisfaction level (Level 1 in Kirkpatrick's assessment model) and motivation level (Level 2a in Kirkpatrick's assessment model). In questions about satisfaction and motivation, the answers were based on a 5-point Likert scale, ranging from 1 = the level/ amount was much higher during face-to-face clerkship than it was during online clinical clerkship and 5 = the level was much higher during online clinical clerkship than it was during faceto-face clerkship.
Total number of respondents and the number of respondents included in the study (Fig 1) For the derivation dataset, we collected responses from 2,640 Japanese medical students. Among the 2,640 respondents, 2,594 (98.3%) consented to participate in this study. The sample size was calculated based on a 99% confidence interval (CI) and a 5% margin of error. The student population enrolled in their fifth and sixth years at Japanese medical universities totaled 18,195. The required sample size was 642, and in this study, the total number of respondents was 2,594, which is 4-fold larger than required. Respondents with missing answers were excluded from the study. Of the remaining respondents, we included those who indicated having experienced a face-to-face clerkship in a hospital prior to the first emergency declaration and then subsequently switching to an online clinical clerkship. Finally, 1,671 responses were used in the derivation analysis (Fig 1).
For the validation dataset, we collected responses from 132 Japanese medical students. Among the 132 respondents, 130 (98.5%) consented to participate in this study. Of the remaining respondents, we included those who indicated having experienced a face-to-face clerkship in a hospital prior to the second emergency declaration and then subsequently switching to an online clinical clerkship. Finally, 106 responses were used for the validation analysis (Fig 1).
To compile the validation dataset, we collected responses from 132 Japanese medical students. Among the 132 respondents, 130 (98.5%) consented to participate in this study. Respondents with answers missing from their profile were excluded from the analysis (n = 0). In Japan, fourth-and fifth-year students are the equivalent years of practice, so we also excluded responses from students in other years (n = 0). The next step was to categorize the responses To compile the derivation dataset, we collected responses from 2,640 Japanese medical students. Among the 2,640 respondents, 2,594 (98.3%) consented to participate in this study. Respondents with answers missing from their profile were excluded from the analysis (n = 127). In Japan, fifth-and sixth-year students are the equivalent years of practice, so we also excluded responses from students in years below the fifth year (n = 19). The next step was to categorize the responses according to the type of education the students received mid-COVID-19. Sixteen respondents were excluded for inappropriate responses. Of the 2,432 remaining respondents, 1,758 reported that they had experienced a face-to-face clerkship in a hospital pre-COVID-19 and then switched to an online clinical clerkship due to the COVID-19 crisis. Of the 1,758 respondents, those with missing responses regarding the content of the education to which they were exposed were excluded (n = 87). Finally, 1,671 responses were used for derivation analysis in this study. https://doi.org/10.1371/journal.pone.0263182.g001 according to the type of education the students received mid-COVID-19. No respondents were excluded due to an inappropriate response. Of the 130 remaining respondents, 107 reported that they had experienced a face-to-face clerkship in a hospital pre-COVID-19 and then switched to an online clinical clerkship due to the COVID-19 crisis. Of the 107 respondents, those with missing responses regarding the content of the education to which they were exposed were excluded (n = 1). Finally, 106 responses were used for validation analysis in this study.

Statistical analysis
We used SPSS v27 (IBM, Armonk, NY, USA). Statistical significance was set at a two-sided pvalue of 0.05.
Using the derivation dataset, we performed a multivariate logistic regression analysis to identify the factors associated with each outcome measurement of the effectiveness of online clinical clerkship, per the outcomes defined in the Measuring Outcomes section. We categorized the outcomes into the following binary: equivalent/better than face-to-face clerkship or not. The predictive variables included (a) lecture duration, (b) lecture frequency, (c) quizzes, (d) assignment submission, (e) student oral presentations, (f) observation of practice, (g) clinical skills practice, (h) participation in interprofessional meetings, (i) interactive discussions with physicians, and (j) technical Internet-related problems. We categorized the answers to (a)-(h) into three categorical variables: more opportunities in face-to-face clerkship, same opportunities in online as in face-to-face clinical clerkship, and more opportunities in online clinical clerkship. We categorized (j) (technical Internet-related problems) into three variables: not at all, a little, and a lot. In this study, we did not incorporate variables related to medical students' profiles because doing so would not have been in keeping with the study's aim to develop a prediction model for the effectiveness of the online clinical clerkship curriculum.
Based on multivariate logistic regression analysis, we only incorporated associated factors (p < 0.1) into the final prediction model. Using multivariate logistic regression analysis, the final model determined point values for each associated variable and developed an integerbased estimation system for satisfaction and motivation, respectively. By applying this model to the validation dataset, we assessed the prognostic instrument's predictive accuracy, with discrimination. Discrimination (i.e., the degree to which a model differentiates between participants answering equivalent/better satisfaction/motivation in faceto-face clerkship) was calculated using concordance (c-) statistics, ranging from 0.5 (no discrimination) to 1.0 (perfect discrimination). Table 1 presents the participants' demographic information and the proportion of respondents that selected each answer to every survey question.

Results
Identifying the factors associated with satisfaction/motivation level regarding online clinical clerkship S1 Table shows

Validation analysis to assess the scoring model's prediction performance
We determined our predictive instrument's discrimination by applying it to the validation dataset. Based on these scores, we created receiver operating characteristic (ROC) curves to Cut-off value 22 17 a: Not at all: 0, less than once a week: 1, once a week: 2, twice a week: 3, three times a week: 4, four times a week: 5, five times a week: 6, six times a week: 7, seven times a week: 8, eight times a week: 9, nine times a week: 10, ten times a week: 11, over ten times a week: 12. https://doi.org/10.1371/journal.pone.0263182.t002 predict whether students' satisfaction/motivation was equivalent/better than in face-to-face clerkship, as shown in Fig 2. The c-statistics, which indicate the area under the curve to predict satisfaction and motivation, were 0.69 � (95% CI: 0.55-0.83, p = .005) and 0.75 � (95% CI: 0.60-0.90, p = .001), respectively. When the cut-off value for the satisfaction scores was set to 22, sensitivity was 0.50, and specificity was 0.89. When the cut-off value for the motivation scores was set to 17, sensitivity was 0.71 and specificity was 0.85.

Discussion
We developed a practical instrument to predict the effectiveness of online clinical clerkship based on medical students' satisfaction and motivation. Although an instrument to predict the online clinical clerkship curriculum's effectiveness has not yet been developed, such an instrument is needed, especially given the increase in demand for online clinical clerkship due to the COVID-19 pandemic.
As the results show, our model's prediction performance was relatively high (satisfaction: 0.69, motivation: 0.75). Satisfaction and motivation are based on individual self-assessment, which means that different people can rate their satisfaction with and motivation for the same curriculum differently. Individual factors (such as sex and year in medical school) are reported to be related to satisfaction and motivation [30,32]. Nevertheless, we did not incorporate individual components into the score in our prediction model. This was because our goal was to develop a scoring model that predicts the effectiveness of online clinical clerkship curricula. In short, our model's prediction performance was high in spite of not incorporating individual demographic information. Using our scoring system, educators will be able to both predict the effectiveness of and improve their online clinical clerkships. Our instrument will be of great help to educators in building a better online clinical clerkship curriculum. In our study, the best cutoff values for motivation and satisfaction were 22 and 17, respectively ( Table 2). Educators can use these scores as guides to improve online clinical clerkship by enhancing the educational approaches in their curriculum, with the goal of exceeding the scores we obtained. For example, to maintain medical students' satisfaction and motivation during online clinical clerkships in a manner that is comparable to face-to-face clerkship, educators might consider providing the students with the same opportunities to observe physicians' practice and to practice their clinical skills as face-to-face clinical clerkship (at only these two variables, score for satisfaction:20, and score for motivation:18) ( Table 2).
This study has several strengths. The assembled population is one of the largest study sample sizes for predicting the effectiveness of online clinical clerkship. Our instruments were developed using big data from national-level cross-sectional studies involving a total of 1,671 medical students in the derivation dataset and 106 students in the validation dataset. Another strength is that our scoring system is a practical instrument that can be applied to the online clinical clerkship curriculum. The predictors are well defined and easily measured, so that students' online opportunities to experience each educational approach can be evaluated as equivalent to, or more or less frequent than in face-to-face clerkship. We also made technical troubles simple to evaluate in three frequency categories: never, occasionally, and often.
Our study has some limitations. First, the cross-sectional study retrospectively measures subjects' exposure and outcome simultaneously. Thus, recall bias was unavoidable. Second, our prediction model is based on medical students' satisfaction and motivation, which are self-assessed outcomes that are easily measured using a questionnaire. However, knowledge and skill acquisition, which are Level 2b in Kirkpatrick's assessment model, are more objective outcomes [26,[29][30][31]. Predictive instruments based on such outcomes, as opposed to satisfaction and/or motivation, might be more useful. Therefore, more research might be required to show the correlation with exam scores or patient outcomes before using this study as a tool to modify curricula. The third limitation is that qualitative factors on the part of the faculty were not considered in the current model because the current study was a quantitative survey. For example, the content and quality of lectures, the faculty's skills of online facilitation might also affect the students' satisfaction and motivation. Forth, in this study, we only included medical students in Japan. Future research should consider the applicability of these results to other countries. Online clinical clerkship would greatly depend on health care resources, Internet infrastructure in countries/communities, and whether students are familiar with the devices required to access online education [33,34]. This indicates that our model would be difficult to apply in countries with limited healthcare resources and/or a lack of Internet infrastructure. Another limitation is that these data are based on online clinical clerkship during the COVID-19 pandemic. The impact of COVID-19 varies across countries and communities. Furthermore, online clinical clerkship post-COVID-19 pandemic may be of a different nature (e.g., a combination of face-to-face and online). We believe that our prediction model can be applied in other countries' online clinical clerkship curriculum or post-COVID-19 pandemic. However, we need to evaluate our model's validity in other countries and post-COVID-19 pandemic in future studies.

Conclusion
We successfully developed a scoring model to predict the effectiveness of the online clinical clerkship curriculum, based on students' satisfaction and motivation. Our scoring model will accurately predict and improve the online clinical clerkship curriculum's effectiveness.